United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent ond Trademark Office 
Address: COMMISSIONER FOR PATENTS 
P.O. Box 1 450 

Alexandria. Virginia 22313-I4S0 
www.usplo.gov 



APPLICATION NO, 


FILING DATE 


FIRST NAMED INVENTOR 


ATTORNEY DOCKET NO. 


CONFIRMATION NO. 


i 0/676,377 


^ 09/30/2003 


Ulrich Neumann 


06666/1 5600I/USC-3345 


3241 



20985 7590 02/13/2007 

FISH & RICHARDSON, PC 
P.O. BOX 1022 

MINNEAPOLIS, MN 55440-1022 



EXAMINER 



AMINI.JAVIDA 



ART UNIT 



PAPER NUMBER 



2628 



SHORTENED STATUTORY PERIOD OF RESPONSE 



MAIL DATE 



DELIVERY MODE 



3 MONTHS 02/13/2007 PAPER 

Please find below and/or attached an Office communication concerning this application or proceeding. 

If NO period for reply is specified above, the maximum statutory period will apply and will expire 6 MONTHS 
from the mailing date of this communication. 



PTOL-90A (Rev. 10/06) 



Office Action SumniQrv 


Application No. 

10/676.377 


Applicant(s) 

NEUMANN ET AL 


Examiner 
Javid A. Amini 


Art Unit 
2628 





-- The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) OR THIRTY (30) DAYS. 

WHICHEVER IS LONGER, FROM THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1.136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MOrfTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent temn adjustment. See 37 CFR 1 .704(b). 

Status 

1)13 Responsive to communication(s) filed on 21 December 2006 . 
2a)l3 This action is FINAL. 2b)n This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 O.G. 213. 

Disposition of Claims 

4) 0 Claim(s) is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) n Claim(s) is/are allowed. 

6) 13 Claim(s) 2-10. 12-13. 15-23. 25-26. 29-31. 33-34. 37-39. 45-47 is/are rejected. 
?)□ Claim(s) is/are objected to. 

8) n Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) n The specification is objected to by the Examiner. 

10)D The drawing(s) filed on is/are: a)[J accepted or b)n objected to by the Examiner 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

1 !)□ The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12)n Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-{d) or (f). 
a)n All b)n Some * c)^ None of: 

1 .□ Certified copies of the priority documents have been received. 

2. n Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 



Attachment(s) 

1) □ Notice of References Cited (PTO-892) 4) □ Inten^iew Summary (PTO-413) 

2) □ Notice of Draflsperson's Patent Drawing Review (PTO-948) Paper No(s)/Mail Date. . 

3) □ Information Disclosure Slatement(s) (PTO/SB/08) 5) □ Notice of Infomial Patent Application 

Paper No(s)/Mail Date . 6) □ Other: . 



U.S. Patent and Trademark Office 

PTOL-326 (Rev. 08-06) 



Office Action Summary 



Part of Paper No./Mail Date 20070208 



Application/Control Number: 10/676,377 Page 2 

Art Unit: 2628 

Response to Arguments 
Applicant's arguments filed 12/21/2006 have been fully considered but they are not 
persuasive. 

Regarding to the Applicant's arguments the rejections under 35 U.S.C. 1 12 are 
withdrawn. 

Applicant on page 19 of the remarks argues that the second reference Moura does not 
generate a three-dimensional model of an environment from range sensor information 
representing a height field for the environment. However, Moura teaches generating a three- 
dimensional model of an object from a two-dimensional image sequence. 

Examiner's reply: Moura teaches generating a three-dimensional model of an object (it 
may be an environment) from a two-dimensional image sequence (emphasis added), it does not 
mean from just one image, it means from a plurality of images. As Moura discloses clearly at 
col. 2 lines 36-51 that according to one embodiment, the system includes: an image sensor for 
capturing a sequence of two-dimensional images of a scene, the scene including the object; a 
two-dimensional motion filter module in communication with the image sensor for determining 
from the sequence of images a plurality of two-dimensional motion parameters for the object; 
and a three-dimensional structure recovery module in communication with the two-dimensional 
motion filter module for estimating a set of three-dimensional shape parameters and a set of 
three-dimensional motion parameters from the set of two-dimensional motion parameters using a 
rank 1 factorization of a matrix. The system may also include a three-dimensional shape 
refinement module to refine the estimate of the three-dimensional shape using a coarse-to-fine 
continuation-type method. 
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Examiner's suggestions: Applicant may emphasis the type of ^^range sensor" used in 
this claimed invention, or may specify the environment considers as a 3-D object. 

Applicant on page 20 regarding claim 22, argues similar to the previous argument. 

Applicant on page 20 regarding claims 29 and 37 argues that the Moura does not obtain a 
3-D model of and environment identifying in real time a region in motion with respect to a 
background image 

Examiner's reply: Copied what Moura at col. 4, lines 63-67 discloses that the filters 18, 
20, 22, 24 in fig. 1 receive digitized images of a video sequence captured by the image sensor 12 
and successively process the original video sequence to extract from it 3D models for the 
background and moving objects that are present in the video sequence. 

Applicant on page 21 argues that Moura does not teach the underlined part of claim 29. 

Examiner's reply: Moura at col. 5, lines 14-31 teaches the concept of the claim language. 
Moura teaches obtaining a three-dimensional model of an object (it may be an environment) 
from a two-dimensional image sequence (emphasis added), it does not mean from just one 
image, it means from a plurality of images. See Moura discloses clearly at col. 2 lines 36-51. 

Moura at col. 4, lines 63-67 discloses that the filters 18, 20, 22, 24 in fig. 1 identifying in 
real time a region in motion with respect to a background image in real-time video imagery 
information from at least one image sensor having associated position and orientation 
information with respect to the three dimensional model, the background image comprising a 
single distribution background dynamically modeled from a time average of the real-time video 
imagery information; Moura at cols. 1-2, lines 66-67 and 1-11, respectively, teaches placing a 
surface that corresponds to the moving region in the three dimensional model; Moura in figs. 12a 
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and 13a illustrates projecting the real-time video imagery information onto the three dimensional 
model, including the surface, based on the position and orientation information; and 
visualizing the three dimensional model with the projected real-time video imagery; Moura at 
col. 4, lines 63-67 discloses that the filters 18, 20, 22, 24 in fig. 1 identifying a region in motion 
in real time comprises subtracting the background image from the real-time video imagery 
information, identifying a foreground object in the subtracted real-time video imagery 
information, validating the foreground object by correlation matching between identified objects 
in neighboring image frames, and outputting the validated foreground object; wherein identifying 
a foreground object comprises identifying the foregroxmd object in the subtracted real-time video 
imagery information using a histogram-based threshold and a noise filter; Moura at col. 8 lines 
15-31 teaches identifying a region in motion in real time further comprises estimating the 
background image by modeling the background image as a temporal pixel average of five recent 
image frames in the real-time video imagery information. However, Moura does not teach 
average of five recent image frames, but as Moura discloses at col. 8 line 29 the larger the region 
the larger the weight estimate, see expression 7. 

For the above reasons the previous rejection is still maintained. 

Claim Rejections - 35 USC § 103 
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 
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Claims 2-10, 12-13, 15, 23, 25-26, 29-31, 33-34, 37-39, and 45-47 rejected under 35 

U.S.C. 103(a) as being unpatentable over Frederick Weinhaus and Venkat Devarajan with title of 

"Texture Mapping 3D Models of Real-World Scenes" ACM Vol. 29, No.4 1997 pp.325-363, 

(hereinafter refers as Frederick), and fiirther in view of Moura et al. US 6,760,48881, 

(hereinafter refers as Moura). 

Claim 12. 

Moura in fig. 4 steps 32, 38 and 42 illustrates the following claim languages: A method 
comprising: generating a three dimensional model of an environment from range sensor 
information representing a height field for the environment; Moura at col. 2 lines 37-38 the 
system includes: an image sensor for capturing a sequence of two-dimensional images of a 
scene. 

Moura at col. 3, lines 4-6 teaches the invention uses a parametric description of the shape 
and the induced optical flow parameterization which has advantages over the prior arts tracking 
feature points that may be unreliable when processing noisy video sequences. As a result, to 
alleviate this situation in the prior art, it is known to assume a very short interval between frames 
for easy tracking. See following part of the claim: tracking orientation information of at least one 
image sensor in the environment with respect to the three-dimensional model in real-time; Moura 
at col. 22 lines 2-8 teaches that the 3D models obtained from the video data according to the 
present invention may be used to build a synthetic image sequence. This synthesis is achieved 
by specifying the sequence of viewing positions along time. The user specifies the viewing 
positions, either in an interactive way or from an automatic procedure. 
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The next step of the claim i.e. projecting real-time video imagery information from the at 
least one image sensor onto the three-dimensional model based on the tracked orientation 
information; Moura at cols. 1-2, lines 66-67 and 1-11, respectively, teaches according to one 
known technique which does not require computing of an estimate of the absolute depth as an 
intermediate step, the 3D positions of the feature points are expressed in terms of Cartesian 
coordinates in a world-centered coordinate system, and the images are modeled as orthographic 
projections . The 2D projection of each feature point is tracked along the image sequence. The 
3D shape and motion are then estimated by factorizing a measurement matrix whose entries are 
the set of trajectories of the feature point projections. The factorization of the measurement 
matrix, which is rank 3 in a noiseless situation, is computed by using a Singular Value 
Decomposition (SVD) expansion technique. 

Moura in figs. 12a and 13a illustrates visualizing the three-dimensional model with the 
projected real-time video imagery. 

Moura for the following section of the claim "wherein projecting the real-time video 
imagery information comprises generating a depth map image from a video sensor viewpoint, 
arid projective texture mapping the real-time video imagery information onto the three 
dimensional model conditioned upon visibility as determined from the generated depth map 
image" clearly in fig. 4 and at col. 26 lines 5-22 teaches the feature points were tracked by 
matching the intensity pattem of each feature along the sequence. Using the rank 1 weighted 
factorization described hereinbefore, the 3D motion and the relative depth of the feature points 
was recovered from the set of feature trajectories. Fig. 15 show two perspective views of the 
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reconstructed 3D shape with the texture mapped on it. The angle between the wails is clearly 
seen and the round edge is also reconstructed. 

Moura is silenced about the limitations in next part of the claim, however, Frederick at 
second column second paragraph on page 349 teaches the claim limitations a An SGI computer 
was used to perform the rendering by simply interpolating the color values across the triangles 
and by using a depth-buffer to remove hidden pixels. See part of the claim's limitations: wherein 
generating the depth map image and projective texture mapping the real-time video imagery 
information are performed using a one-pass approach on graphics hardware that supports SGI 
OpenGL extensions. 

Frederick is silenced to have one image sensory comprises multi image sensors. 

Thus, it would have been obvious to one ordinary skill in the art at the time the invention 
was made to combine Moura's fig. 17 step 24 into Frederick's texture mapping for an efficient 
and less computationally-intensive technique to recover 3D structure fi-om a 2D image sequence. 
Claim 2. / 

Frederick on page 355 at first col. teaches that in this system, 3D models composed of 
parametric primitives such as blocks, pyramids, and the like are manually composited to 
approximate the geometry of the architectural structure using the imagery as a visual guide. 
Claim 3. 

Frederick in fig. 3c illustrates the claim limitations. 
Claim 4. 

Frederick on page 356 at second col. teaches that retrieving the most appropriate oblique 
perspective image stored in the video disk bank and then warp it according to Equation (1) to 
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simulate the pilot's or sensor's view of the gaming area at that instance in time. Since these 
operations were accomplished in real-time, it would give the illusion of smooth interactive 
motion through the gaming area. 
Claim 5. 

Frederick on page 342 at first col. teaches the height-field approach also has been applied 
to the urban setting where the elevations of the terrain posts are raised locally to account for the 
heights of buildings. Frederick on page 344 at the bottom of first col. teaches under section 3.2.2 
Point Projection Approaches for Height Fields. 
Claim 6. 

Frederick on page 349 at second coL teaches in Hughes' approach, only those terrain 
elevation tiles that were found to lie at least partially within the output image's ground 
"footprint" (projection onto the X, Y ground plane) were rendered. The terrain elevation model 
and imagery resolutions were selected as a function of distance between the terrain elevation tile 
and the eye point so that coarse resolution could be used for more distant regions. Terrain 
elevation tiles were then tessellated into triangular meshes at the specified resolution. 
Claim 7. 

Frederick on page 355 at first col. teaches it may even be moved along the ray from the 
eye point to the object. Shadow mensuration is an option that can be used to help in 3D model 
construction. Also, shadows can be rendered into a graphic overlay on the transformed view. 
Semi-automated 2D feature extraction tools, such as road following and the detection of 
boundaries of buildings whereby the user starts the extraction process with a manual cue, also are 
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included. SRI also has experimented with automatic extraction of 3D models of buildings and 
used their system to portray the results. 
Claims 8-9. 

Frederick on page 327 at first col. under section (4) teaches the claim limitation. 
Frederick on page 326 at second col. teaches Volotta interactive video has put together a system, 
called the Mars Navigator. 
Claim 10. 

Frederick on page 340 at the bottom of first col. teaches multiple views. 
Claun 13. 

Frederick on page 352 at second col. teaches a combination of automatic stereo 
compilation followed by manual editing can be used to generate the terrain elevation model. 
Claim 22. 

See rejection of claim 12 that applies to the rejection for claim 22. 
Claim 15. 

Frederick on page 340 at the bottom of first col. teaches multiple views. 
Claim 16. 

Frederick on page 360 at first col. teaches the claim limitation. 
Claim 17. 

The claim limitation is obvious because when generating a 3D image, the viewpoint of 
the video imagery projection should be separate fi-om viewpoints fi-om multiple image sensors. 
Claim 18. 

Frederick in fig. 8 illustrates the claim limitation. 
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Claim 19. 

Frederick on page 342 at first col teaches the height-field approach also has been applied 
to the urban setting where the elevations of the terrain posts are raised locally to accoimt for the 
heights of buildings. Frederick on page 344 at the bottom of first col. teaches under section 3.2.2 
Point Projection Approaches for Height Fields. Frederick on page 349 at second col. teaches in 
Hughes' approach, only those terrain elevation tiles that were found to lie at least partially within 
the output image's ground "footprint" (projection onto the K ground plane) were rendered. 
The terrain elevation model and imagery resolutions were selected as a function of distance 
between the terrain elevation tile and the eye point so that coarse resolution could be used for 
more distant regions. Terrain elevation tiles were then tessellated into triangular meshes at the 
specified resolution. 
Claim 20. 

Frederick in fig. 8 illustrates the claim limitation. 
Claim 21. 

Frederick on page 355 at bottom of first and top of the second columns teaches a novel 
view-dependent texture-mapping technique is used to render the architectural model. Multiple 
photographs are projected onto the model in order to texture its surface completely. (An image- 
space shadow-mapping algorithm based on a z-buffer is used to keep track of obscurations.) 
However, since the photographs overlap, the rendering algorithm must decide which photograph 
or photographs to use at each output pixel. 
Claim 23. 
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Frederick on page 339 at first col. teaches in addition to showing transformed images in 
perspective, panoramic, and orthographic formats, they also presented a pair of transformed 
images that could be viewed stereoscopically in 3D. Frederick on page 355 at bottom of first and 
top of the second columns teaches a novel view-dependent texture-mapping technique is used to 
render the architectural model. Multiple photographs are projected onto the model in order to 
texture its surface completely. (An image-space shadow-mapping algorithm based on a z-buffer 
is used to keep track of obscurations.) However, since the photographs overlap, the rendering 
algorithm must decide which photograph or photographs to use at each output pixel. 
Claim 29. 

The rejection of claim 29 is similar to the rejection of claim 12. 
Claim 25. 

Frederick in fig. 8 illustrates the claim limitation. 
Claim 26. 

Frederick in fig. 4 illustrates the claim limitations. 
Claim 30. 

Frederick on page 355 at second col. teaches a weighted average of the textures fi-om the 
overlapping images is used, where the weights are the angular deviations of the viewing vectors 
of each source image fi:om that of the output view. Moreover, to avoid visible "seams," the 
weights are ramped near the bovmdaries of source photographs. Optionally, a model-based stereo 
correlation algorithm may be used to refine the 3D model to include finer detail such as recessed 
windows, and the like, and to increase the fidelity of the texture-mapped rendering. 
Claim 31. 
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Frederick in fig. 8 illustrates the claim limitation. 
Claim 37. 

See rejection of claim 12 that applies to the rejection for claim 37. 
Claim 33. 

Frederick in fig. 8 illustrates the claim limitation. 

Claim 34. 

Frederick in fig. 4 illustrates the claim limitations. 
Claim 38. 

Frederick on page 327 at first col. under section (4) teaches the claim limitation. 
Frederick on page 326 at second col. teaches Volotta interactive video has put together a system, 
called the Mars Navigator. 
Claim 39. 

Frederick on page 342 at first col. teaches the height-field approach also has been 
applied to the urban setting where the elevations of the terrain posts are raised locally to account 
for the heights of buildings. Frederick on page 344 at the bottom of first col. teaches imder 
section 3.2.2 Point Projection Approaches for Height Fields. Frederick on page 349 at second 
col. teaches in Hughes' approach, only those terrain elevation tiles that were found to lie at least 
partially within the output image's ground "footprint" (projection onto the A^, K ground plane) 
were rendered. The terrain elevation model and imagery resolutions were selected as a fimction 
of distance between the terrain elevation tile and the eye point so that coarse resolution could be 
used for more distant regions. Terrain elevation tiles were then tessellated into triangular meshes 
at the specified resolution. 
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Claim 45. 

Frederick in table 1 teaches the claim limitation. 
Claim 46. 

Frederick on page 338 teaches the selected portions of some of these frames 
corresponding to face-on views of building facades were mapped onto the faces of 3D models of 
the buildings. Distant mountains were also textured to add a sense of realism. Computer 
synthesized images simulating driving down the streets as well as low altitude views were then 
generated in segments and stored on video disk for later playback under joystick control. During 
playback, turns could only be initiated at selected locations such as street intersections. 
Claim 47. 

Frederick in table 1 teaches the claim limitation. 

Conclusion 

THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1 .136(a), 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1.136(a) will be calculated fi-om the mailing date of the advisory action. In no event, 
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however, will the statutory period for reply expire later than SIX MONTHS from the mailing 
date of this final action. 

Any inquiry concerning this communication or earlier conununications from the 
examiner should be directed to Javid A. Amini whose telephone number is 571-272-7654. The 
examiner can normally be reached on 8-4pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Kee Tung can be reached on 571-272-7794. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for impublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would 
like assistance from a USPTO Customer Service Representative or access to the automated 
information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 



Javid A Amini 
Examiner 
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